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Abstract 

The Investigator studied three Doctor of Philosophy program concentrations for a two year 
period, 2013-2015. The intent of the confidential study was to determine factors affecting 
program participants' perceptions most. A mixed methods embedded study was applied, with the 
quantitative portion representing the secondary data source and the qualitative portion primary. 
Questionnaires were provided to the doctoral students and 42 of 165 were completed and 
received. Quantitative results revealed average to above average program satisfaction, but a 
substantial range of perceptions regarding quality of the program, connectedness to advisors and 
other students, most useful courses, and least useful courses. Qualitative results revealed 27 areas 
of concern possessed by the study participants regarding their doctoral journeys. Of the 27, four 
were selected for further analysis: concerns regarding too few students completing the 
dissertation process, concerns with lack of meaningful formative and summative assessments 
occurring across courses, retribution among some faculty for students expressing opinions 
regarding courses and their programs, and faculty and advisor turnover. Follow up interviews of 
both study participants and program faculty, beyond the initial survey, provided greater 
illumination regarding the four selected items. Study results can be applied to the particular 
program, in the spirit of continuous improvement. The results can also be considered by others 
designing program evaluations of participant perceptions of education programs. The merit of 
using mixed-methods embedded study design in program evaluation became obvious. 

Keywords'. Doctoral Programs, Doctoral Students, Division I University, Minority 
Students, Graduate Records Examination, Open Enrollment, Student Perceptions, Program 
Evaluation, Retention, Persistence, Mixed Methods Research, Embedded Design, Confidential 


Studies 
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Student perceptions of their doctoral programs: case study 

What are the perceptions of doctoral students regarding their programs? How do these 
perceptions affect them, and their potential success, in programs? Does it matter how students 
perceive their doctoral programs? Yes? No? Why? Why not? 

From 2013 to 2015, the Investigator studied a Doctor of Philosophy degree program at a 
southern Division I university in the United States. The degree program houses three 
concentration majors and many minors. There are approximately 175 students enrolled. A 
majority of the students represent minority races. 

The Students 

While potential doctoral program students are required to take the Graduate Record 
Examination (GRE), the test is used to determine if students admitted to the program must take 
remedial courses in writing and reasoning skills. They must also possess at least a 3.0 grade point 
average (g.p.a.) in their previously obtained Master’s degree. Master’s degrees, universally, have 
such a g.p.a. requirement, as a condition for receiving the degree. 

Basically, the doctoral program is one where all students can be accepted, regardless of 
GRE normative percentile rankings. Students with the lowest possible 1 percentile GRE rankings 
have been admitted. The intent of admitting students across all GRE scores is to empower those 
student groups who have been disenfranchised historically. The university’s undergraduate 
population is two-thirds minority as well. Comparable universities graduate three times as many 
undergraduate students. 

The University and Doctoral Faculty 

The university faced financial difficulties 30 years ago. A President was hired who 
focused on attracting local area minority students. Most of these students were first generation 
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college students. There was a substantial minority population from which to increase 
enrollments. Using this model, the university experienced remarkable growth and financial 
success. Today, the university continues to attract a majority minority student population, many 
of whom are first generation college students. There is also a considerable international student 
population. Several new programs and colleges have been added to the university, including the 
Doctor of Philosophy program. 

The Doctor of Philosophy (PhD) program was begun 13 years ago. The leader of a 
university affiliated organization identified a Professor to be hired. An offer was made by the 
leader to the founding Professor, without a formal search, and the program began. Shortly 
thereafter, the leader of the university affiliate who hired the founding Professor joined the 
doctoral faculty. A Mathematics professor was also added to the doctoral faculty, for the sake of 
ensuring coverage of the statistical side of the research courses and studies. Then, a former 
colleague of the founding Professor was attracted to the program. 

The university had campuses in three foreign countries. For political reasons, one of the 
foreign countries removed many U.S. universities from its borders. As a result, two former 
directors of that location were ultimately placed as faculty in the main campus-based PhD 
program. One of the two had earned their doctoral degree, while directing the out of country 
university program, from a branch campus of a midwestem regional comprehensive university. 
The second person, upon return to the United States, entered the university’s PhD program, 
completed it, and was hired as faculty in it. Then, a sixth person was hired who had completed 
their doctoral degree from a southern regional comprehensive university. Ultimately, a 
qualitative expert was needed among the doctoral faculty. A person was attracted from another 
doctoral program in the state to fill this need. 
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Two years prior to this study, the founding Professor suddenly died. The year prior to this 
study the leader of the university affiliated organization resigned to head another entity in the 
university. In the second year of this two year study, two of the aforementioned remaining 
faculty announced their retirements. A third professor, who was in their second year on faculty, 
replacing the deceased founding Professor, announced their resignation. A fourth member of the 
faculty of seven was named a Dean of another college within the university. Thus, four of seven 
doctoral faculty members departed at the end of the second year of this study. 

A search for an additional doctoral faculty member was conducted in the second year of 
the study. However, the search failed. Two offers were made, but both declined. One new 
Professor was hired, beyond the failed search, who worked in another of the university’s 
departments, without a formal search. At the beginning of the study there were seven doctoral 
faculty members. At the end of the study there were four. Obviously, there was considerable 
faculty turnover in the PhD program, in a short period of time. 

With these student, university, and doctoral faculty contexts and realities in mind, what are 
the perceptions of the doctoral students in the programs? What prior studies have been done on 
doctoral student perceptions that can inform this one? What has been previously measured in 
these studies? What has been previously revealed regarding doctoral students’ perceptions of 
their programs? What else needs to be revealed? 
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Literature Review 

While prior studies of doctoral student perceptions of their programs are sparse, there are 
a few in the recent past. The study assumption is doctoral students’ perceptions of their programs 
impact their performance in, and satisfaction with, their programs. 

An early study on this topic focused on what level has the greatest impact on doctoral 
student perceptions most. Is it the university, the graduate school, or the particular programs? 
Apparently it is the program, rather than the university or graduate school. The particular 
department program, not the graduate school or university, contributes to doctoral students’ 
perceptions of progress and success most (Ehrenberg, 2009). 

With this in mind, further research was done that focused on perceptions of particular 
programs. What is it about particular doctoral programs that affect student perceptions most? The 
Ehrenberg study findings led to research regarding the satisfaction levels of doctoral students 
across various academic disciplines and different institutional types (Barnes & Randall, 2012). 
This study further affirmed that student perceptions are affected most by the particular program, 
instead of the college, school, or university. While doctoral students desire to be graduates of 
highly ranked doctoral programs, the greatest impact on the students’ perceptions are the 
particular programs of which they are a part. 

Doctoral student perceptions, informing satisfaction, also contribute to their completion 
of programs. Student retention, persistence, and graduation rates have become very high stakes 
for programs and universities, in the wake of tightening accreditation rules and decreases in 
potential student populations. 

Low feelings of doctoral student-to- doctoral student and doctoral student-to-faculty 
connectedness in the learning environment may be predictive of departure from doctoral 
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programs (Terrell, Snyder & Dringus, 2009). In other words, if doctoral students feel connected 
to other students in their programs, as well as faculty, they are more apt to persist and remain in 
them. 

The sense of relatedness to the doctoral student’s advisor positively affects student 
perceptions regarding doctoral programs, as do students’ autonomy (freedom to do one’s own 
research) and competency (perceived ability to complete the doctoral program) (Mason, 2012). 

Doctoral students’ lives beyond their programs also have an effect on their perceptions of 
programs. Challenges faced by doctoral students outside of their programs can greatly impact 
their progress in them (Moffett, 2006). Historically, little attention has been paid to the life needs 
and realities of doctoral students beyond their programs. This lack of attention is especially 
detrimental to international doctoral students in the United States, especially those students who 
are culturally distant. 

While prior studies included identification of institutional level affecting perceptions 
most, along with connectedness to faculty, advisor, and fellow students, they did not include 
students’ perceptions of most and least effective courses in their programs, and most and least 
difficult courses. They also did not feature doctoral students’ perceptions of effective and 
ineffective instruction in their courses and programs. Additionally, prior studies were not 
designed in a mixed methods embedded format. 

In addition to foci of past studies incorporated in this study, it was determined by the 
Investigator that asking study participants to identify most useful and least useful courses, and 
most and least difficult courses, in their programs would be beneficial. Also, data regarding most 
and least effective instruction would be useful and included. Additionally, the Investigator 
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concluded, providing an open ended response prompt could yield perceptions beyond those from 
the focused prompts and questions from previous studies and this one. 

As mentioned, prior studies did not utilize a mixed methods embedded design. Of the 
several mixed methods designs, an embedded design was most appropriate for this study. The 
embedded design is a mixed methods design in which one data set provides a supportive, 
secondary role in a study based primarily on the other data type (Creswell, et al, 2003). In this 
study’s case the primary data would be the qualitative short answer responses, and follow up 
interviews of some study participants and program faculty, resulting from the short answer 
responses. The supportive, secondary portion of the study would be the quantitative portion of 
the survey instrument. 

It was also determined that a confidential, rather than anonymous, survey response 
instrument be used. The confidential design would enable the Investigator to follow up with 
particular respondents as needed. This follow up would enhance the selected, primary qualitative 
portion of the study. Such follow up to respondents’ short answer responses on the survey 
instrument had the potential to further illuminate participant perceptions of their doctoral 
programs. Interviewing some of the program faculty would enhance triangulation of study 
results, as well. 

The current gaps in the research include doctoral students’ perceptions of quality of 
particular courses in their program, the quality of instruction, open-ended, free-response 
opportunities for study respondents, and the mixed methods embedded study design. 
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Study Results 

The Investigator distributed hard copy study questionnaires, Institutional Review Board 
(IRB) disclosure statements, and self addressed stamp envelopes for same to study participants 
across student meetings, in doctoral program classes, and via U.S. mail from May, 2014 through 
August, 2014. One hundred sixty-five study packets were distributed. Forty two study 
questionnaires and signed copies of IRB disclosure forms were completed and mailed to the 
Investigator. Therefore, the completed study questionnaire return rate was 25 percent. 

The quantitative ranking results, serving as the secondary data source are as follows 
(note: survey items 1, 2, 3, 8, 9, and 10 featured ranking opportunities for respondents): 


1. On a scale of 1 to 10 (with “1” feeling not connected whatsoever and “10” feeling 
completely connected), how connected do you feel to your program advisor? 

Score Mean: 7.71 Score Range: 9 Standard Deviation: 2.50 

2. On a scale of 1 to 10 (with “1” feeling not connected to other students in the 
program and “10” feeling completely connected to other students in the program), 
how connected do you feel to other students in the program? 

Score Mean: 7.15 Score Range: 9 Standard Deviation: 2.61 

3. On a scale of 1 to 10 (with “1” feeling not connected to the faculty in the program 
and “10” feeling completely connected to the faculty in the program), how 
connected do you feel to the faculty in the program? 

Score Mean: 7.23 Score Range: 8 Standard Deviation: 2.26 

8. On a scale of 1 to 10 (with “1” feeling as though you have no “freedom” whatsoever 
to conduct your own research and “10” feeling as though you have total “freedom” 
to conduct your own research), how much “freedom” do you feel you have to 
conduct your own research? 


Score Mean: 8.71 Score Range: 5 Standard Deviation: 1.59 
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9. On a scale of 1 to 10 (with “1” feeling as though you are not competent whatsoever 
in completing your doctoral program and “10” feeling fully competent to complete 
your doctoral program), how competent do you feel in completing your doctoral 
program? 

Score Mean: 8.79 Score Range: 9 Standard Deviation: 1.67 

10. One a scale of 1 to 10 (with “1” not feeling challenged whatsoever by other life 
responsibilities beyond your doctoral program and “10” feeling completely 
challenged by other life responsibilities beyond your doctoral program), how 
challenged do you feel by other life responsibilities beyond your doctoral program? 

Score Mean: 8.04 Score Range: 9 Standard Deviation: 2.45 
Quantitative Ranking Results Discussion 

The mean score range across the six prompts was 7.15 to 8.79, suggesting average to 
above average participant perception satisfaction with the program concentrations. However, 
there was a substantial score range across the six prompts, with four of them exhibiting a range 
of 9, and a fifth with a range of 8. But, despite the substantial score ranges, overall quantitative 
results revealed average to above average perceived student satisfaction with the program 
concentrations. Historically, quantitative evaluations of programs have been utilized most. Had 
this study only used a traditional quantitative approach, the results would suggest satisfactory 
participant perception results. 


Courses in the Programs 

Survey questions 4, 5, 6, and 7 had to do with courses in the programs. Below are the 
quantitative results of doctoral students’ opinions regarding most beneficial course, least 
beneficial course, most difficult course, and least difficult course (note: due to the large number 
of courses identified in response to each question, the courses with the largest frequency of 


responses in each category, along with the number of responses= “n” for each, are shared): 
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4. Of the courses you have taken in your doctoral program, which course has been 
most beneficial to you? 

Research Methods and Tools (n=7) 

History and Philosophy of Higher Education (n=5) 

Qualitative Research Design (n=5) 

5. Of the courses you have taken in your doctoral program, which course has been 
least beneficial to you? 

Concepts of Leadership (n=4) 

Social Science Statistics with SPSS (n=4) 

Organizational Assessment and Survey Administration (n=3) 

Qualitative Research Design (n=3) 

6. Of the courses in your doctoral program, which course has been the most difficult 
for you? 

Social Science Statistics with SPSS (n=14) 

Qualitative Research Methods (n=4) 

Concepts of Leadership (n=3) 

Research Methods and Tools (n=3) 

7. Of the courses in your doctoral program, which course has been the least difficult 
for you? 

Ethics for the Professions (n=7) 

Belief Systems (n=4) 

Concepts of Leadership (n=4 

Courses in the Program Discussion 

Quantitative participant perception results regarding courses in the program revealed a 
variety of courses being most beneficial, least beneficial, most difficult, and least difficult. The 
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quantitative results merely show numbers of participants’ perceptions about particular courses. 
Qualitative student responses regarding most/least beneficial and most/least difficult courses 
provide more particular details regarding the categories. 

Interestingly, the course Concepts of Leadership appeared in both most difficult and least 
difficult courses. The course is one that has been taught by four of the seven doctoral faculty in 
the recent past. Perhaps the perceptions of it being both most and least difficult are the result of 
several different instructors teaching it across semesters. This could suggest different course 
designs, and expectations possessed across instructors, for the doctoral students. Further 
discussion regarding the program courses will be shared in the qualitative portion of the study 
results. 

Qualitative Survey Results 

The qualitative student response results, serving as the primary data source in the mixed 
methods embedded study design, are as follows: 

For survey items 1, 2, 3, 8, 9, and 10 featured ranking opportunities for respondents (note: for 
these responses the most positive and most negative responses are shared): 

Item #1 regarding connectedness to Program Advisor: 

Positive Responses: 

Dr. XXXXXXX is always available and provides support and guidance in an engaging and 
personal way. After our meetings I truly feel that all my concerns have been addressed with 
individual attention. 

I have always had a great rapport with my advisor. They were supportive and offered the 
guidance I needed to complete my studies. 

My advisor has been & is an amazing mentor, guide, & counselor. I have felt completely safe & 
comfortable discussing my challenges & concerns with them. More important they have always 
put my best interests first & ensured increased & enhanced opportunities for learning & growth! 
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Negative Responses: 

About a year ago my advisor was changed. I have not had the opportunity to take a class with 
them yet, and they are primarily in a different concentration, so I do not feel very connected. 

There have only been two brief meetings with my advisor; an insufficient amount of time to 
build any kind of rapport. 

They’re not exactly an “outreach” kind of person. 

Item #2 regarding connectedness to other students in the program: 

Positive Responses: 

I feel extremely connected with other students in the program. The diversity with the doctoral 
students, age, ethnicity, occupations, & cultural background has led me to have friendships & 
academic learning experiences that have really enriched me. 

I have built a tight knit group of friends in the program we’ve taken classes together, studied 
together, & shared a big part of lives with each other. I know that they will be a part of my life 
forever. 

We work in teams; we care for each other genuinely (regardless of the concentration). Due to 
small classes, events, bringing food, working collaborative the students get to form strong 
friendships. I can truly say I have made friends for a lifetime (our friendship is strong) and I have 
friends from all over the world! 

Negative Responses: 

Prior to completing my course work, I had very few contacts with peers in the program. After 
completing course work on the program it was very hard to know where any one of us were. In 
fact, I don’t even know if anyone of us has graduated since after the comprehensive exam was 
taken. 

I have a real job & family that prevented me from playing school. In many ways, it was much 
like high school part two. Lots of in-group out-group dynamics. 

There is very little connection with other students since most of them are working and also the 
structure of the program does not promote any connection or relationship among students. The 
courses should be offered according to whether one is 1st, 2nd, or 3rd year - that way students 
get to know each other as they are taking class together. 

Item #3 regarding connectedness to faculty in the programs: 


Positive Responses: 
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Overall I felt close to the faculty. The faculty I interacted with offered a great deal of support and 
assistance throughout the program. 

I feel as if all faculty members are genuinely concerned for the doctoral students & are willing to 
go the extra mile to support students & their needs. The spirit of the institution fortifies feelings 
of connectedness to the faculty in the program. I have only had classes w/ about half of the 
faculty. 

I feel that they are not only professors but also mentors. If we need any assistance with research 
they are always willing to help. 

Negative Responses: 

Meet with faculty only in necessary situations, (class, meeting, symposium. . .) 

No “home” program affiliation. No doctoral mentoring. No outreach. I will be more aggressive 
in pursuit of support. 

If I do not take any of their classes there is no connection whatsoever. There should be 
connection leading to growth or depth in the relationship of faculty and students from the 
application stage throughout the course of study. The dissertation writing should be part of the 
journey not the summit. 

Item #8 regarding freedom to do one’s own research: 

Positive Responses: 

I have found the faculty to be supportive in helping us find research that is a good fit for our 
skills, interests and resources as well as making a significant contribution to the global research 
community. I feel completely supported in my decision to explore a mathematics education 
construct. 

I feel that I have full control on what I want to research. 

My advisor has never curbed my research ideas. I think the transformative nature of this 
program/my concentration is evidenced in the fact that my dissertation has nothing to do with 
technology in education which is my experiential background! 

Negative Responses (note: there was only one response to this question judged to be negative): 

At the beginning of my dissertation, I was literally given to Dr. XXXXXXX. It’s not their fault, 
but they threw my proposal away. We were not a good fit and now I’m with a different chair & 2 
yrs were wasted. 

Item #9 regarding feeling competent to complete the doctoral program: 
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Positive Responses: 

I am certain that I will successfully complete the program. I have complete confidence in my 
instructors and my ability to do excellent work toward completion. 

All tools & resources are available to complete my doctoral program. It is just simply a matter of 
time before it is completed. 

I feel highly competent in completing my doctoral program because of the instructor’s genuine 
care and concern for students. In short, instructors help students become self-directed and 
constructive by building on and expanding student’s knowledge base. 

Negative Responses: 

I feel this way because this is the 3rd time I began my proposal, but I feel better about my 
situation. 

Some days, I feel as though I’m in over my head while there are others when I feel as though I 
can accomplish anything. 

I feel I have the competency, but am lacking connection to the program. Frankly, I’m 
floundering. 

Item #10 regarding feeling challenged by other life challenges beyond the doctoral 
program: 

Least Challenged Responses: 

I was not employed during the time I was in the doctoral program. It was possible for me to give 
undivided attention to my studies. 

I am an international student and full time student. I get financial assistance from sources without 
much conditions. 

Most Challenged Responses: 

This is by far the most difficult degree I’ve earned not only because of the time and financial 
investment but because it demands that I redefine myself. I’m a different person now. When 
faced with previously paralyzing challenges, I now build support teams and craft solutions - all 
with an amount of confidence and faith that I did not possess prior. I am a horribly shy person 
and the journey has shoved me completely out of my comfort zone. I’m incredible humbled to 
have the opportunity to work with and learn from amazing people from all over the world. I 
knew this would be difficult but I put in all into perspective. Working multiple jobs while raising 
my children was something I simply committed to with my whole heart. I approached this 
project in much the same way. Form an emotional perspective, it has been one of the very few 
mostly positive experiences I’ve had in my life. 
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I have other responsibilities apart from my doctoral studies. These other responsibilities limit the 
amount of focus I would love to give to my studies. 

I choose to take one class a semester to help lessen those challenges; however, it is still difficult 
to take a class, raise a child, and work full time. 

Analysis of Qualitative Responses for Survey Items 1, 2, 3, 8, 9, and 10 

Qualitative narratives provided by the 42 study respondents were analyzed and organized 
by survey item. The most positive and negative responses for each survey were featured in the 
results, in the spirit of providing contrasts for each prompt. 

Regarding connectedness to Advisor, positive responses included having concerns 
addressed with personalized attention, great rapport and support, amazing mentoring, guidance, 
and counseling. Negative responses included too many changes of Advisors, too few meetings 
with the Advisor, and the Advisor not being approachable. 

Regarding connectedness to other students in the program, positive responses included 
being enriched by the friendships and learning experiences, building a tight knit group of friends 
in the program, and working in a team; as a result of small classes, events, and bringing in food 
to classes. Negative responses included not knowing where peers were in the dissertation process 
beyond the doctoral course work, having life obligations beyond the program that kept students 
from making the needed student-to-student connections, and the structure of the doctoral 
program being such that it does not promote meaningful peer connectedness; meaning, that the 
program should be more of a cohort model with students grouped together in year one, year two, 
year three, etc. 

Regarding connectedness to faculty in the program, positive responses included a great 
deal of faculty support and assistance throughout the program, genuine concern displayed by 
some faculty members, and a feeling of some faculty being not only professors but also mentors. 
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Negative responses included only interacting with faculty during “necessary situations;” class, 
meetings, symposiums, no “home” program concentration affiliation, no mentoring, and no 
connection whatsoever with particular faculty members if classes are not taken with them. 
Further: 

“There should be connection leading to growth or depth in the relationship 

of faculty and students from the application stage throughout the course of study. 

The dissertation writing should be part of the journey not the summit.” 

Regarding freedom to do, and conduct, one’s own research, positive responses included 
faculty being helpful in finding a good fit for student research skills, interests and resources, 
possessing full control on what is researched, and never being curbed regarding research ideas. 
The only negative response for this prompt discussed being assigned to a faculty dissertation 
guide and their misplacing the students’ proposal, resulting in a loss of two years work. 

Regarding feeling competent to complete the doctoral program, positive responses 
included feeling certain that the program can be successfully completed, having all tools and 
resources necessary to complete the program, and some instructors causing students to become 
self-directed by building the needed knowledge base. Negative responses included being in the 
third attempt of beginning a dissertation proposal, feelings of being in over one’s head, and 
lacking needed connectedness to the program resulting in floundering. 

Regarding feeling challenged by life issues beyond the program, least challenged 
responses included not having other work responsibilities beyond the doctoral program, and 
having needed financial assistance so energies could be entirely focused on the program. Most 
challenged responses included challenges with the time and financial commitment to the 
program, having other life responsibilities that adversely affect performance in the program, and 
taking a half load in the program as a result of the life responsibilities beyond the program. 
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Most/Least Beneficial and Most/Least Difficult Courses Qualitative Results 
Item #4 regarding most beneficial course (single selected responses): 

Belief Systems (n=9) 

The word culture only meant “cultural sensitivity” to me as taught in many work related 
seminars. Learning about cultural themes opened a new perspective to me & pointed me toward 
future research topics/questions. 

Research Methods and Tools (n=7) 

Research Methods & Tools gave me the academic overview of the research process necessary as 
a PhD student. 

History and Philosophy of Higher Education (n=5) 

This course, although intensive reading is required, allows ample time for the student-led 
discussions. That is doctoral quality work. 

Qualitative Research Design (n=5) 

Dr. XXXXXXX is an excellent professor who knows this subject and is very competent and 
motivating. They demand excellence and I appreciate their standards. 

Item #5 regarding least beneficial course (single selected responses): 

Concepts of Leadership (n=4) 

The way the course was structured made it difficult to make course material applicable. 

Social Science Statistics with SPSS (n=4) 

I do not think that this course provided an adequate overview of introductory statistical methods. 
I also do not believe that it prepared me enough for the advanced statistics course. 

Organizational Assessment and Survey Administration (n=3) 

Not nearly enough instruction on survey development, administration, & analysis given that this 
is a MAJOR research tool. 

Qualitative Research Design (n=3) 

While I have taken away something from each course that the others could not have offered, I 
chose this course because we were given a special topic. I would have benefited more if I were 
allowed to choose my own research topic. 
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Item #6 regarding most difficult course (single selected responses): 

Social Science Statistics with SPSS (n=14) 

There are many quantifiable approaches to cover in class and not enough time to fully explain 
them. Once (I thought) I learned one it was time to move to the next. Only course where I needed 
and had a tutor. 

Qualitative Research Methods (n=4) 

The professor was not at all clear of what was expected of us in the course. Research strategies 
taught were not designed properly, student learning outcomes were not there at all. 

Concepts of Leadership (n=3) 

In any reading intensive course, interaction with my professor and classmates is critical. The 
classroom dynamic in this course was very negative. This was an environment where I felt that 
my professor had little respect for me and I did not like the activities they assigned the class as 
they created this exclusionary atmosphere. I just didn’t have a strong sense of self-efficacy 
consequently still don’t feel as if I’ve sufficiently learned the concepts. 

Research Methods and Tools (n=3) 

Because so much of the information was new. For example, I had never done a lit review before 
this class and the process was painful, not only because I wasn’t sure of what I was doing but 
also because the review had to be written as part of a team and my partner was an ESL student. 

Item #7 regarding least difficult course (single selected responses): 

Ethics for the Professions (n=7) 

I consider ethics for the professions a critical part & requirement for all employees, students, 
peers & collegial members. This class was something I embraced so fully that overall content 
seemed easy. 

Belief Systems (n=4) 

I think that this course could have challenged my way of thinking a bit more, especially since it 
was a course on culture. However, I did find the concepts I learned to be very beneficial to my 
program of study. 

Concepts of Leadership (n=4) 

I believe the structure of the class gave me that peaceful easy feeling 

Analysis of Qualitative Results for Most/Least Beneficial and Most/Least Difficult Courses 
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The most beneficial results of courses included enhanced cultural sensitivity, an overview 
of the research process, an understanding of doctoral level course work, and an appreciation for 
high expectations and standards. 

The least beneficial results of courses included lack of coherence of course materials, the 
lack of coherent instruction for the complexities of data analysis software, lack of needed 
instruction on survey development, administration, and analysis, and lack of freedom of selected 
research topics in a course. 

The most difficult courses featured too much to cover in class and not enough time to 
fully explain it, lack of clarity of what was expected of students in the course, improper research 
design strategies on the part of the instructor, lack of identifiable course outcomes, negative 
classroom dynamics on the part of the instructor, lack of respect for students resulting in a 
weakened sense of student efficacy, and overemphasis on creating literature reviews as course 
product that were not related to student needs. 

The least difficult courses featured being one with the course’s content and naturally 
thriving in it, and the lack of being challenged in positive ways. 

Open-Ended Response Item 

11. Please provide any additional information you would like to share regarding your 
perceptions about the doctoral program: 

Positive Responses 

This is a great program and I am delighted to associate with instructors, colleagues and the 
university. I feel very fortunate to be in the program. 

It is a great program in general. The learning/teaching community is wonderful. Great full time 
faculty. It is hard work but I am competitive & enjoyed academia. I would certainly recommend 
it to other students. I love the university! The Department of Education is unique. I would want 
to see more teaching/research assistantships available & more scholarships/financial aid. The 
Graduate Research Office is super excellent support! For the most part I love it so far. 
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I appreciate the fact that the program is geared toward student success and degree completion. 
Critical Responses 

More emphasis should be placed on “writing for publication” & “Grants.” I do not understand 
why the life blood of a researcher is not a core course. Knowing where funds come from & how 
they function is key. Publication goes without saying. 

Generally the program is great & has challenged me to continue to grow as a scholar & person. 
One concern that I have is that a number of students are getting stuck in the proposal process. 

I’m not too sure where the disconnect is during this phase of the program. 

A more directional degree plan should be developed. The degree plan itself should guide the 
students throughout the semesters - year-wise, semester- wise, a logical sequential and 
corresponding process of registering for courses. Present degree plan is only the list of courses, 
not helpful. 

A rational and more sensible comprehensive examination methods need to be developed. After 
completing all course works, in doctoral level, students should be evaluated and assessed - on 
understanding of ideas, concepts, and philosophy; their application in context of reality & life; 
not how much and what they remembered! 

In current practice, after completing 54 credit hours, in about three years time, testing a student 
for eight hours in one day does not justify assessment of understanding and applications; even, 
perhaps not fair to students. 

Analysis of the Qualitative Open Response 

Positive open responses included the program being great, the learning/teaching 
community being wonderful, the school being unique, and the program being geared toward 
student success and degree completion. 

Negative open responses included the program needing to emphasize writing for 
publication and grant writing more, concerns with the number of students who are getting stuck 
in the dissertation proposal process, the need for a more directional degree program; meaning, 
“year-wise, semester-wise, a logical sequential and corresponding process of registering for 
courses”, a more rational and logical doctoral comprehensive examination, and elimination of the 
“8 hour seat time comprehensive exam.” 
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Final Analysis of Initial Qualitative Results 

Without qualitative results, student perceptions regarding the program appeared to be 
satisfactory. Quantitatively, the program is rated by students as average to above average. 
Satisfactory program outcomes, while pleasing, do not provide the needed opportunities for 
program improvement. Continuous program improvement is becoming more high stakes, as 
accrediting bodies are emphasizing this type of program data-driven assessment design. 

Opportunities for improving the program reside most in the students’ negative feedback 
to survey questions. Therefore, the Investigator focused on areas of student concern. The areas of 
greatest student concern included: 

-Too many changes of Advisors, too few meetings with the Advisor, and the Advisor not being 
an “outreach” kind of person. 

-Not knowing where peers were in the dissertation process beyond the doctoral course work. 

- Having life obligations beyond the program that kept students from making the needed student - 
to-student connections. 

-The structure of the doctoral program being such that it does not promote meaningful peer 
connectedness. 

- Being assigned to a dissertation guide and the guide misplacing the student’s proposal, 
resulting in a loss of a year’s work. 

- Being in the third attempt of beginning a dissertation proposal. 

- Feelings of being in over one’s head. 

-Lacking needed connectedness to the program resulting in floundering. 

-Challenges with the time and financial commitment to the program. 

-Having other life responsibilities that adversely affect performance in the program. 
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- Only being able to carry a half load in the program, as a result of the life responsibilities 
beyond the program. 

- Lack of coherence of course materials. 

-Lack of coherent instruction for the complexities of data analysis software. 

-Lack of needed instruction on survey development, administration, and analysis. 

-Lack of freedom of selected research topics in a course. 

- Too much to cover in class and not enough time to fully explain it. 

-Lack of clarity of what was expected of students in the course. 

-Improper research design strategies on the part of the instructor. 

-Lack of identifiable course outcomes. 

-Negative classroom dynamics on the part of the instructor. 

-Lack of respect for students resulting in a weakened sense of student efficacy. 

-Overemphasis on creating literature reviews as course product that were not related to student 
needs. 

- The lack of being challenged in courses in positive ways. 

- The need to emphasize writing for publication and grant writing more. 

- Concerns with the number of students who are getting stuck in the dissertation proposal 
process. 

-The need for a more directional degree program; meaning, “year- wise, semester-wise, a logical 
sequential and corresponding process of registering for courses”. 

- The need for a more rational and logical doctoral comprehensive examination. 


Follow up Interviews 
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Because the study was confidential, rather than anonymous, the Investigator was able to 
seek more clarity regarding the items of greatest concern from study participants. Several study 
participants were asked follow up questions in interviews, regarding the areas of greatest 
concern. 

In the follow up interviews study participants expressed concerns about the number of 
students who successfully completed the doctoral program of studies, but were unable to succeed 
in the dissertation portion of the degree program. 

Students also expressed concerns about the number of courses that merely featured a final 
course project literature review. They reported that literature reviews were turned in at the end of 
the semesters and no feedback regarding them were received. No other assessments were evident 
in the courses. Participants were not sure why they received the grades they did in these courses, 
since they received no feedback regarding what they had done. They would often receive the end 
of course literature review assignment with a grade but no feedback. 

Several study participants reported many of the doctoral students shared that they did not 
complete the questionnaire for this study, due to a general mistrust of some doctoral faculty, and 
a commonly held belief that if they were honest about the program there would be retribution on 
the part of some of those faculty. Although the Investigator had informed potential respondents 
of the confidential nature of the study, along with providing the IRB disclosure, many students 
were concerned about the consequences of their sharing feelings about courses and the program. 
When asked further about this perceived lack of trust, several of the students shared incidences 
where students had spoken up about issues in courses and the program concentrations. They 
reported students who had expressed these concerns were subsequently punished in arbitrary 
ways by some of the doctoral faculty. 
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Students reported concerns about the number of advisors they had while in the program, 
the number of faculty who came and went during their time in the program, and the quality of 
product of some of the students in the program. 

Follow up Interviews Results 

Study participants expressed concerns about the number of students who were unable to 
successfully complete the degree program, the lack of formative and diagnostic assessments in, 
and across courses, the lack of feedback regarding summative assessments in, and across, 
courses, retribution among some of the doctoral faculty for students expressing concerns about 
courses and the program, and program advisor and faculty turnover. 

Interviews With Program Faculty 

As a result of the follow up student interview results, the Investigator opted to interview 
two of the seven doctoral faculty members. When the Investigator asked about study 
participants’ perceptions regarding concerns about students being able to complete the doctoral 
program, and achieving candidacy, but unable to complete dissertations, the following was 
shared: 

“The doctoral program has basically been one with open enrollment. Students with GRE 
percentile rankings as low as 1 percentile have been admitted. There are competing 
interests between the university’s faculty requirements for scholarship and service and 
meeting the needs of the doctoral students. Candidates are provided great liberty in 
selecting dissertation topics, and often their topics do not coincide with research interests 
and required scholarship needs of faculty.” 

When the Investigator asked about the doctoral students’ perceptions of some faculty only 
assigning literature reviews, collected at the end of courses and not providing needed feedback, 


the following was shared: 
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“While the university claims the school is one where the emphasis is on teaching, the 
scholarship requirements and tenure process are such that requirements are more 
reflective of a high research university. There is an institutional disconnect between what 
is expressed and what is measured. In order for faculty to remain on successful tenure 
track, the first priority must be to produce publishable research and substantial service. If 
faculty expends the needed work energies to assess the doctoral students as they should in 
courses, there is simply not enough time to meet the scholarship and service 
requirements. It comes down to whether or not faculty can remain in employment, due to 
university demands, when they decide to only require an end of the course assignment. It 
is a survival technique.” 

When the Investigator asked about doctoral student perceptions regarding there being retribution 

for students speaking out about courses and programs, the following was shared: 

“There are some faculty members who are arbitrary with the students. It is unfortunate. In 
one case a faculty member’s tenure nomination was voted down by the tenured faculty of 
the school, due in part to their treatment of students. The tenure process here includes 
tenured faculty within the school first voting on potential tenure for faculty. Then, the 
university council considers the vote and can either agree with the school’s 
recommendation or not. In the case of this one professor, a high-ranking university 
official overrode the tenured school faculty’s vote and convinced the university council to 
override the school’s vote and tenure them. This decision was not in the best interest of 
the students and program. Additionally, two other faculty members worked for the 
university in other capacities prior to becoming professors in the doctoral program. They 
understand the university’s emphasis from above is about getting students into the 
program to enjoy the needed revenue. Their perception is the revenue is more important 
than meeting student needs. Frankly, it is a business of dollars and cents. For these 
reasons, they know how things have always been done and perpetuate this status quo. 
They resist change. The revenue generated from the program is the priority. These are 
examples of the politics within the organization that ultimately negatively impact the 
students.” 

When the Investigator asked about the doctoral students’ perception of there being turnover of 

advisors and faculty in the program, the following was shared: 

“There has been considerable faculty and advisor turnover in the doctoral program the 
past few years, as a result of a death, retirements, and now resignation. But the public 
commentary by the program leadership focuses on faculty retirements. There has been no 
public discussion regarding faculty resignation. I suppose this is to quash any potential 
discussions regarding faculty, or student, dissatisfaction with the courses, concentrations, 
and programs. The governance model has been one where the consensus model for 
making decisions has allegedly been in place. What has actually been in place was a 
particular faculty member with university-wide authority making the decisions. It was 
called a consensus model because other faculty knew to agree with that person. Other 
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faculty followed that person’s lead due to their universal power and authority. However, 
when that faculty member left the program the so called consensus model continued. But 
there was no longer a faculty member with particular power and authority in the group. 
When a faculty member introduced a motion, utilizing Robert’s Rules, in a meeting, there 
was shock and resistance by those who had been followers of the old model. Other 
faculty said they did not know Robert’s Rules. Ironically, they all were participants in 
Robert’s Rules in the larger School of Education monthly meetings. So, their feigned lack 
of knowledge of Robert’s Rules was dishonest. They simply were resisting and blocking 
change. The faculty member who introduced it was surprised by fellow faculty’s 
dishonesty and attempted deception. The faculty could not agree on shifting away from 
the alleged consensus model which impeded progress. There was great resistance to 
procedural and substantive change in any forms. Repeatedly, the old guard among the 
faculty stated, “We have always done it this way. . .” For this reason, and others, faculty 
departed from the program.” 

Conclusions, Summary, and Recommendations 
The study revealed 27 areas of student perceived concerns. Four of the areas were 
selected for further research. The areas prompted follow up student and program faculty 
interviews. The four areas were: concerns with students completing the dissertation process of 
the degree program, lack of formative and summative assessments in courses, retribution for 
students speaking out about problems with courses and the program, and advisor and faculty 
turnover in the program. 

The Investigator concluded the open enrollment nature of the doctoral program was 
related to the substantial numbers of Candidates who are unable to complete the dissertation 
portion of the degree. If the doctoral program admits anyone who possesses the minimum 
required grade point average for a completed Master’s degree, and who simply takes the GRE, 
then it must meet the many unique needs of students entering, and traveling through, the 
program. 

The inability to complete dissertations was exacerbated by the lack of formative and 
diagnostic assessments across courses, as well as the lack of feedback for the summative 
literature reviews. The faculty scholarship realities are such that they compete with meeting the 
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program’s unique students’ needs. Doctoral Candidates should be guided in their research areas 
so it better complements the university scholarship demands of the professors, while causing 
graduates of the doctoral program to select topics that enhance their employability. In other 
words, there is too much freedom in Candidate selection of dissertation research topics. 
Contemplating the control of freedom of doctoral student research, for students’ sake, as well as 
faculty’s, needs further research, as the Investigator previously did not consider student freedom 
to do research as a possible detriment. 

Students not having their academic needs met was compounded by faculty being tenured 
by the university who did not receive a majority vote within the school to be tenured. Retribution 
on the part of some faculty is most likely connected to those who mistreat the students, and enjoy 
being tenured by the university against the will of others in the school who are tenured. The 
university should revisit the policy and honor the tenured faculty decision votes within the 
school. 

Summary 

Traditional quantitative evaluations of programs provide limited, and often desirable, 
results. In the case of this study, the quantitative portion of the mixed methods design, used as 
the secondary data source, yielded average to above average doctoral student perceptions of their 
program. This result suggested participant satisfaction with the program. However, when the 
qualitative portion of the embedded design was analyzed, a substantial number of areas of 
concern were discovered. These additional data results, featuring 27 areas of concern, provide 
opportunities for program improvement. Without the mixed methods embedded design approach, 
such opportunities for improvement would not have been discovered and be possible. 


Recommendations 
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The Investigator recommends the people with power and authority in the doctoral 
program studied review the students’ 27 areas of concern, address them in the spirit of 
continuous program improvement, and make the needed program changes and improvements. 

Beyond the study results, the merit of the mixed methods embedded design, with the 
qualitative component providing the primary data, seems obvious. Further research involving the 
application of mixed methods embedded research, with the qualitative portion serving as the 
primary data source, is needed. 

In this age of intensified data driven analyses of programs, provoked by enhanced 
accreditation rules; not to mention doing the right thing, the application of the mixed methods 
embedded study design seems appropriate for authentic program evaluation and improvement. 

As evidenced in this study, the design provides opportunities for program evaluators to discover 
participants’ perceptions of areas of concerns, beyond ordinary satisfactory results. 

There will be resistance to going beyond usual satisfactory data results. Great energies 
are expended in preserving programs as they are, by many parties involved in them. While 
continuous program improvement sounds like a positive thing, it can be painful to those 
responsible for the changes. Program changes can also be difficult for the students. 

Utilization of study designs, that unearth traditionally hard to find areas of concern, 
should be prefaced with cautions to upper echelons of college administrations, so as to not cause 
the results to be misinterpreted as inadequacies on the part of administrators of those programs. 
But, first, the administrators of programs must be willing to buy in to uncovering concern areas. 
Again, for many reasons, this will be difficult. Life is easier for all concerned when programs are 


perceived to be satisfactory. 
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There can be a fine line between perceptions of poor performance by program directors, 
and coordinators, and mixed methods embedded design program evaluation assessment 
uncovering areas of concern. Mixed methods embedded study results can set off proverbial 
program quality alarms, unnecessarily. All parties concerned with continuous improvement of a 
program should understand that a program may well be satisfactory but still have many areas of 
concern. It is the areas of concern, and acknowledging and addressing them, which will take the 
programs to new places. 

The discovered areas of concern are opportunities for program improvement. Program 
improvement has potential for making a program all it can be. However, acknowledging 
formerly hidden areas of concerns in programs are ripe with complex political issues, and 
therefore highly sensitive. There will be great resistance to this new program evaluation 
approach, for reasons mentioned. 

More research is needed in how to best promote program readiness for the realities of 
discovering areas of concern, in seemingly satisfactory programs, to reduce potential tensions 
among program and institutional decision makers, when such results are realized and shared. It is 
vital for programs to go beyond merely producing desirable data results. 

The goal can no longer be to deliver data results that merely suggest a satisfactory 
program. Programs can no longer afford to do things as they have always been done. Instead, 
efforts must be made in the area of utilizing program evaluation study designs, resulting in 
discoveries of program concerns that have been historically hidden; or, worse yet concerns that 


have been historically ignored. 
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In other words, we can no longer afford to apply study designs that merely yield 
satisfactory program results. It is no longer business as usual in higher education. Instead, we 
must do what is not customary and traditional, and do what is difficult. 

We must apply study designs that yield areas of concern, in programs that otherwise 
appear to be satisfactory. As program evaluators, we must dare to seek and share the program 
concerns. Then, in the spirit of continuous program improvement, those with the power and 
authority to do so should address the program concerns and genuinely improve them. 
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